Method and device for temporal filtering of disparity maps

ABSTRACT

The invention consists in a method of and a device for temporal filtering of disparity maps of n different frames, each map indicating the disparity of pixels of 3D images comprising the steps of marking the stationary pixels and the non stationary pixels and implementing temporal filter for stationary pixels, detecting limitations for temporal filtering corresponding to a variation of moving pixels above a determined threshold value and automatically de-activating the temporal filters as response to the detected limitations for temporal filtering.

1. FIELD OF THE INVENTION

The invention concerns temporal filtering of disparity maps of 3D imagesand more precisely the deactivating or reactivating of temporalfiltering. It concerns 3D rendering.

2. PRIOR ART

Disparity calculation is an essential tool in computer vision or in the3D industry. It is used in numerous applications such as robotnavigation through depth ranging, gesture recognition, viewinterpolation for creating multi-view content from a stereo image pair,2D to 3D conversion.

Disparity is typically calculated from two views with a differentparallax, like the left and right camera from a stereo rig. It consistsin a correspondence search for identical features in both images andmeasures their relative position difference in both images.

Hence a problem that arises out of processing stereo sequences is thetemporal variation of disparity measurements. Filtering technique,referred as “temporal filter” is capable of reducing the temporalvariations of disparity measurements. This technique improves thedisparity matching.

One of the difficulties of the disparity matching is further thetemporal inter frame stability. A disparity match in low textured areasof an image pair can differ from one frame to the other, in particularin areas where there are neither movements, nor illumination variation.The bad result here from is that those areas exhibit temporal flickeringbecause of this almost random variation of the disparity.

Document “temporal filtering of disparity measurements” by L. Di Stefanoet al. discloses a temporal filtering technique which improves temporalconsistency of disparity measurements by reducing the matching errorsdue to the noise affecting the imaging system. Transition“static=>dynamic” implies that the order of the filter must drop to zeroas soon as motion is detected in order to get rid of the past history ofthe point under examination.

For moving objects the filter is then by-passed (i.e. the order of thefilter is set to 0) so as to promptly follow the structural disparityvariations in the scene. Conversely, for static points the order of thefilter is kept as high as possible, through a smooth incrementalvariation, in order to provide the maximum capability to filter awayuncertain disparity measurements. But abrupt changes which are differentfrom static to dynamic or dynamic to static change relate touncontrolled change in the image and are source of disturbing andinterframe instability.

In the FIG. 1, the bottom pictures represent the images from the leftcamera at two different time frames. (Right images are not shown). Onthe top, the disparity maps which are calculated between the left andright images at the two different timeframes are shown as heat maps. Onsome non moving areas that should have the same disparity value, andhence the same color, some differences appear. At least three of themare shown on the maps (that are emphasized with the use of circles).These differences are normal in a matching because in low texturedareas, the matching is not very precise. Then, a temporal lowpass filteris effectively implemented for stationary pixels in a real-time matchingsoftware, taking advantage from the time aspect, in order to make thematching more robust and non fluctuating over time.

But there is some use case limitation of that temporal filtering. Theproblem to be solved by the invention is how to improve the temporalfiltering in some use case limitation.

3. SUMMARY OF THE INVENTION

One particular embodiment of the invention consists in a method oftemporal filtering of disparity maps of n different frames, each mapindicating the disparity of pixels of 3D images comprising the steps ofmarking the stationary pixels and the non stationary or moving pixelsand implementing temporal filter for stationary pixels. The methodfurther comprises the step of

-   -   detecting limitations for temporal filtering corresponding to a        variation of moving pixels above a determined threshold value;    -   automatically de-activating the temporal filters as response to        the detected limitations for temporal filtering.

The invention will permit to implement the temporal filtering of thedisparity map in a general application not limited to the use case.

According to an aspect of an embodiment of the present invention themethod further comprising the step of automatically reactivating thetemporal filters if there is no more limitations for temporal filtering.

In an advantageous embodiment of the invention, detecting limitationsfor temporal filtering consists in detecting if the percentage of pixelsof the disparity maps that have moved within n successive frames isabove a determined threshold value.

In an advantageous embodiment of the invention, the step of detectinglimitations for temporal filtering consists in detecting a scene cut orin detecting accidental cameras moving or in detecting abnormaldisplacement of an object.

In an advantageous embodiment of the invention, the temporal filteringis implemented with a temporal lowpass filter for stationary pixels.

In an advantageous embodiment of the invention, the temporal filteringis implemented with a trilateral filter for non-stationary pixels.

In an advantageous embodiment of the invention, the filter de-activatesautomatically if the percentages of pixels that have that have movedwithin n successive frames is above 15 to 25%.

The invention also relates to a device for temporal filtering ofdisparity maps of n different frames, each map indicating the disparityof pixels of 3D images comprising the steps of marking the stationarypixels and the non stationary pixels and implementing temporal filterfor stationary pixels. Such device comprises means for detectinglimitations for temporal filtering corresponding to a variation ofmoving pixels above a determined threshold value and means forautomatically de-activating the temporal filters as response to thedetected limitations for temporal filtering.

In an advantageous embodiment of the invention, the device comprisesfurther means for automatically reactivating the temporal filters ifthere is no more limitations for temporal filtering.

In an advantageous embodiment of the invention, the means for detectinglimitations for temporal filtering detects if the percentage of pixelsof the disparity maps that have moved within n successive frames isabove a determined threshold value.

In an advantageous embodiment of the invention, the means for detectinglimitations for temporal filtering detects a scene cut, accidentalcameras moving or abnormal displacement of an object.

In an advantageous embodiment of the invention, the temporal filtersde-activate automatically if the percentage of pixels that have movedwithin n successive frames is above 15 to 25%.

4. LIST OF FIGURES

The above and other aspects of the invention will become more apparentby the following detailed description of exemplary embodiments thereofwith reference to the attached drawings in which:

FIG. 1 discloses two disparity maps at different times that havedifferences on some areas, while these areas were stationaries (i.e nomovement occurs in such areas).

FIG. 2 is a work flow in accordance with an exemplary embodiment of thepresent invention.

5. DETAILED DESCRIPTION OF THE INVENTION

Hereinafter, the present invention will be described more fully: First,“temporal filtering” is a technique employed to decrease the temporalvariation of the zones which were badly matched.

On certain parts of the disparity map corresponding to stationary pixelsthere is an operation of local low-pass temporal filtering. Moreover, onother parts of moving pixels temporal filtering includes also atrilateral space filtering of moving pixels. The first step forimplementing the temporal lowpass filter is the detection of stationaryareas. For that purpose actual frame and the immediate neighboring onesrestricted to an odd number are used, for instance, t−2, t−1, t0, t+1,t+2 or 5 frames. For every pixel of the input RGB (Red, Green, Blue)left image, the RGB absolute mean differences D between two successiveframes is calculated as following:

D=Σ _(C={R,G,B})ll_(c)(X, y, t)−l_(c)(x, y, t−1)  I

If the sum D of absolute difference of the intensity of light l_(c) ateach pixel for each color between two successive frames t and t−1 isgreater than a threshold, the pixel is labeled as not stationary ormoving. This is done for all 4 intervals between t−2 and t+2 and thelabels are accumulated. Those pixels will undergo a spatial bilateral ortrilateral filtering.

Finally, only the pixels that have never received a not stationary labelduring the four successive evaluations will be considered stationary,and only those pixels will undergo a temporal low pas filtering.

In a second step, the L-R matching of n frames will be realized. ThisL-R matching produces typically n (odd number) disparity maps for theleft view and as many for the right view.

As already all the pixels that have not undergone a change over the oddn frames have been detected, those stationary pixels can be temporallylow pass filtered in the disparity map for the time t₀ by (for instance)averaging the disparity at that location with the disparities calculatedat frames t−2 through t+2.

The disparity at pixel locations that have been marked as “moving” inone of the n frames can be kept as they have been calculated at t₀. Butit is not very satisfactory to keep them, since those zones are veryvisible in the disparity maps in contrast to those that have beentemporally low pass filtered. Therefore, those zones will undergo atrilateral filtering with a mask. The mask consists in accumulating thepositions of all the pixels that have been moved in one of the n frames.

The trilateral filter with masking will use the disparity map at t₀, theRGB image at t₀ and will hence smooth the disparity over the entireregion where something has moved. The general trilateral filter is usedto filter the noise in the non stationary area detection.

It will be preceded incrementally, by adding the next L-R frames anddoing again stationary pixel detection and temporal low pass filteringon the last n frames.

This known processing represented by the first and second step of FIG. 2is very effective in stabilizing the disparity values in temporallyflickering zones. But this processing has some limitations.

Limitations are per example:

-   -   If there is a scene cut, this will cause the algorithm to        believe that between the frame before the cut, and after the        cut, all pixels have moved, which is not bad but moving pixels        will continue to be searched over n frames which is a time        consuming task on one hand, and on the other hand, the        trilateral filtering will try to even out the differences        between totally different images which ends up with a big mess.        The result is that (n−1)/2 frames before and after the cut will        get totally false disparities for all pixels.    -   If the acquisition system, which consists in two cameras, is        panning, zooming, or if the cameras are moving, a major part of        the pixels will be detected as non stationary and again, the        trilateral filter will try to build some coherence between parts        of scenes where there is none.    -   If everything is fixed (no zooming, panning or camera        displacements), but if a big object in the scene undergoes a big        displacement from one frame to the other, we will end up with        the same problem as described in the previous item.

Therefore, these temporal filters have to be restricted to use casesthat exclude at least the previous three items corresponding toimportant variation of moving pixels.

The invention consists in a way of implementing these temporal filters(the lowpass filter or the trilateral filter) which takes advantages ofits important, useful and very effective ability to make the disparityconstant and robust temporally, without its disadvantages.

FIG. 2 is a work flow in accordance with an exemplary embodiment of amethod of the present invention. The method of temporal filtering ofdisparity maps of n different frames consists in a first step in markingthe stationary pixels and the non stationary pixels. Not-stationarypixels are further called moving pixels. The method consists then in afollowing step in implementing temporal filter for stationary pixels.Furthermore, as represented by the steps 3 and 4 of FIG. 2, theinvention detects the limitation situations described above, and as soonas they have been detected, the temporal filtering will be de-activated.

And if again the use-case is met where the temporal filters areactivated, the temporal lowpass filtering will be re-activated as thetrilateral spatial filtering.

There is a way of detecting the limitation cases to the filtering. As wehave marked all stationary pixels over n frames, we can calculate thepercentage of pixels that have moved within the n frames. If thatpercentage is above a threshold, one of the cases, or all possiblecombinations of them, is arising, and the temporal filteringde-activates automatically. In the case of stationary pixels, only thetemporal low pass filter was activated and will then de-activatesautomatically. As soon as the percentage of moving pixels over n framesis again below the threshold, the temporal filters reactivate. This hasproven to be a very efficient implementation of temporal low passfiltering. It is functional in the matching algorithm Matchbox. We donot need to take care of the special cases and situation, as it detectsautomatically when it can activate the low pass filtering, and when ithas to be de-activated or re activated.

And it detects automatically when it can activate the trilateralfiltering, and when it has to be de-activated or re activated.

Some practical values: 3, 5 or 7 frames can be used to temporally lowpass filtering. Five frames is a typical number in which a pixel ismarked as stationary between two consecutive frames if the RGB absolutemean difference D between two successive frames is below 10 (R, G, and Bcoded from 0 to 255). The temporal lowpass filter de-activatesautomatically if the percentage of pixels that have changed over naccumulated frames is above 15 to 25%.

1. Method of temporal filtering of disparity maps of n different frames,each map indicating the disparity of pixels of 3D images comprising thesteps of marking the stationary pixels and the non stationary pixels andimplementing temporal filter for stationary pixels; wherein the methodfurther comprises the step of detecting limitations for temporalfiltering corresponding to a variation of moving pixels above adetermined threshold value and automatically de-activating the temporalfilters as response to the detected limitations for temporal filtering.2. Method of temporal filtering according to claim 1 wherein it furthercomprises the step of automatically reactivating the temporal filters ifthere is no more limitations for temporal filtering.
 3. Method oftemporal filtering according to claim 1 wherein the detectinglimitations for temporal filtering consists in detecting if thepercentage of pixels of the disparity maps that have moved within nsuccessive frames is above said determined threshold value.
 4. Method oftemporal filtering according to claim 1 wherein the step of detectinglimitations for temporal filtering consists in detecting a scene cut,accidental cameras moving or abnormal displacement of an object. 5.Method of temporal filtering according to claim 3, wherein temporalfilter de-activate automatically if the percentage of pixels that havemoved within n successive frames is above 15 to 25%.
 6. Device fortemporal filtering of disparity maps of n different frames, each mapindicating the disparity of pixels of 3D images comprising the steps ofmarking the stationary pixels and the non stationary pixels andimplementing temporal filter for stationary pixels; wherein the devicecomprises means for detecting limitations for temporal filteringcorresponding to a variation of moving pixels above a determinedthreshold value and means for automatically de-activating the temporalfilters as response to the detected limitations for temporal filtering.7. Device according to claim 6 wherein the device comprises furthermeans for automatically reactivating the temporal filters if there is nomore limitations for temporal filtering.
 8. Device according to claim 6wherein the means for detecting limitations for temporal filteringdetects if the percentage of pixels of the disparity maps that havemoved within n successive frames is above a determined threshold value.9. Device according to claim 6 wherein the means for detectinglimitations for temporal filtering detects a scene cut, accidentalcameras moving or abnormal displacement of an object.
 10. Deviceaccording to claim 8, wherein the temporal filters de-activateautomatically if the percentage of pixels that have moved within nsuccessive frames is above 15 to 25%.